fix: normalize dictionary types in Arrow scans#3444
Conversation
|
Btw, could you please add Closes #3260 (uncommented) to the start of your description? |
|
Reopening this PR — I closed it by mistake. |
this was done, marking as resolved |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
2 participants
Closes #3260
Rationale
Iceberg treats Arrow dictionary encoding as an encoding detail rather than a separate logical type. However,
ArrowScan.to_tablecurrently concatenates batches without decoding dictionary-encoded columns first. A table containing both plain strings and dictionary-encoded strings therefore fails to scan withArrowTypeError: Unable to merge.This can occur in production when files written with dictionary encoding are later rewritten by Athena or Trino optimization into plain strings.
Changes
ArrowScan.to_table.Attribution
I checked the issue and PR history before opening this PR. I did not find an earlier PR or implementation to cherry-pick for #3260.
Verification
make lintmake test(3711 passed, 1534 deselected)uv run python -m pytest tests/io/test_pyarrow.py -q -k "mixed_dictionary or ensure_non_dictionary"